Evaluating Statistical Validity of Research Reports: a guide for managers, planners, and researchers

She also holds a master's degree (1979) in statistics and a doctorate (1983) in biostatistics from Berkeley. This report was prepared under a research contract while the author was a doctoral student. Preface Just as some managers are more professional, effective, and skilled than others, so are some researchers more competent than others in their use of statistics. And, just as managerial decisions have many dimensions, so do statistical decisions. However, decisions of a midlevel manager are subject to scrutiny of superiors only, but statistical decisions in a research report are subject to the scrutiny of all who read it. Consumers of research reports must not just accept conclusions , but must investigate the methods used to obtain them. I urge resource managers and planners to read research reports critically and to judge the choice of statistical method. In this report I explain the need to be critical and describe some ways in which conclusions can be evaluated. In addition, I recommend discussing the statistical validity of research reports with professional statisticians before applying the results to managerial or planning decisions. Inappropriate statistical methods, as well as appropriate methods inappropriately used, can lead to incorrect conclusions in any research report. Incorrect conclusions may also be due to the fact that the research problem is just hard to quantify in a satisfactory way. Publication of a research report does not guarantee that appropriate statistical methods have been used, or that appropriate methods have been used correctly. Publication also does not guarantee that actual measurements are reasonably close to the underlying concept. You may not be able to tell if an appropriate statistical method was used correctly. You can, however, (with the help of a professional statistician) judge whether the choice of method was appropriate. You can also judge how close the actual measurements seem to the underlying concept that the researcher is studying. Two steps are preliminary to judgments about whether the choice of method was appropriate. The first involves categorizing the study according to its primary statistical purpose in terms of examining variables. This purpose may be describing variables, testing hypotheses about variables, exploring relationships among variables, or building prediction models using variables. The statistical purpose of most studies of visual quality has been exploring relationships or building models. The second preliminary step to evaluating the validity of a research report involves categorizing variables according to their level of measurement. …